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Abstract 

We consider the joint design of forwarding policies and controllers for wireless 
control loops where sensor measurements are sent over an unreliable multi-hop wire- 
less network to the controller node. For fixed sampling rate of the control loop, the 
co-design problem separates into two well-defined and independent subproblcms: 
transmission scheduling for maximizing the delay-constrained reliability and opti- 
mal control under packet loss. We develop optimal solutions for these subproblems 
and show that the optimally co-designed system can be found by a one-dimensional 
search over the sampling interval. Several numerical examples illustrate the power 
of our approach. 

Keywords. Linear optimal control; Communication networks; Optimality. 

1 Introduction 

Networked control has been an active area of research for more than a decade (e.g. [16] 
and the references therein) and the literature is by now rather extensive. The research has 
mainly focused on control design methods for a given system and high-level abstraction of 
the communication network in terms of its latency or loss. State-of-the-art control design 
techniques are very powerful when the control system is able to cope with the network 
deficiencies. When the resulting closed-loop performance is unsatisfactory, however, 
they typically do not provide any insight on how the communication system should be 
modified to yield better performance. For instance, it is not immediately clear if a shorter 
sampling interval is better if it also results in a higher packet loss rate in the network. 
On the communication side, current networking protocols tend to focus on maximizing 
the long-term average throughput or to minimize the energy cost of communication, 
and do not provide any direct ways to influence control-relevant network performances 
such as individual packet latency. In fact, little is known about which loss-latency 
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Figure 1: Our co-design framework separates the system design into two well-defined 
subproblems that admit optimal solutions: deadline-constrained maximum reliability 
routing which characterizes the achievable loss-latency pairs and the design of linear- 
quadratic Gaussian controllers under latency and loss to minimize the control perfor- 
mance loss. 

combinations that can be achieved for a single packet transmitted over an unreliable 
multi-hop network, and only very recently, researchers have started to address real-time 
communication over unreliable networks. 

In wired control systems, where sensor data and actuator commands are transferred 
over a high-speed bus, extensive latencies and losses are often due to interaction with 
other traffic. Hence, co-design issues do not appear until the system is very highly loaded 
and can often be dealt with simply by adding an additional communication bus. The 
situation for low-power wireless multi-hop communications is very different, as latencies 
and packet losses appear already in unloaded networks and the use of capacity expansion 
is non-trivial since the wireless medium is shared with other installations. Hence, co- 
design issues arise already for single-loop controllers. 

This paper proposes a co-design framework for single-loop wireless control systems 
that is provably optimal. By parameterizing the system design in terms of the sample- 
time of the digital control loop, the co-design problem separates into two well-defined 
subproblems: to schedule the multi-hop network to maximize the deadline-constrained 
reliability, and to design a controller with optimum performance under (independent) 
packet losses. Drawing on our recent work on deadline-constrained scheduling [26, 31J, we 
demonstrate how the network scheduling problem can be solved to optimality and how 
this characterizes the achievable loss-latency region for the multi-hop wireless network. 
Similarly, for a given communication latency and loss probability, we develop extensions 
to the work by Schenato et al. [23J that allows to compute optimal controllers and 
estimate the associated closed-loop performance, see Figure [T] Finally, we prove that 
adjusting the sampling rate and solving these subproblems allows to find the jointly 
optimal co-designed system. Numerical examples illustrate the power and flexibility of 
our framework. 
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2 Background and related work 



A well-designed co-design framework requires insight and understanding of networking 
and control and how they interact. To gain analytical insight into how the network 
behavior influences the overall system performance, we first consider a simple scenario 
where the state of a stochastic linear system 

dx(t) = ax(t)dt + dw(t) 

should be estimated continuously at a remote location based on periodic noise-free sam- 
ples x(kh) of the state. The aim is to maintain a state estimate x(t) that minimizes the 
loss 



1 f T 

J e = lim sup - / E {(x(t) - x(t)) 2 } dt . 

T->oo T Jq 



Samples are transmitted from the sensor to the estimator over an unreliable channel 
where packets experience a stochastic delay D. If the delay exceeds one sampling inter- 
val, the transmission is aborted and the packet is considered lost. This gives rise to a 
networked control system with both a time- varying delay and packet losses. As shown 
in [18], it is then possible to explicitly characterize the optimal loss 

7 - F r>Di (1 - p)(e 2ah - 1) 1 
e X 4a 2 /i(l - pe 2ah ) 2a 

provided that pe 2ah < 1. This expression reveals that the achievable loss depends both 
on the sampling interval h, the loss probability p and the complete delay distribution 
(not only its mean). Specifically, J e is monotone increasing in h and p, as well as in the 
mean delay and the delay variance. 

However, in communication systems, latency and loss can typically not both be made 
small. The basic mechanism for dealing with lost packets is to use retransmissions, which 
then take additional time. Specifically, consider communication over a single link and 
assume that the medium is divided into time slots of length t s . A single time slot allows to 
transmit one packet, and a transmission attempt fails with probability p s . Assuming that 
h = kt s and that unacknowledged packets are retransmitted in the next time slot, the 
probability that the packet arrives with delay nt s equals (1— p s )p^~ 1 and the probability 
that the packet is not transmitted until the end of the sampling interval is p = p$ . Note 
that to guarantee a small loss probability we must allow for more retransmission attempts 
and hence a longer sampling interval. Unfortunately, for this specific scenario there is 
no interesting co-design: J e is bounded if p^e 2akts = (p s e 2ats ^ k < 1, so if we cannot 
stabilize the estimation error variance for h = t s , then we cannot do so no matter how 
many retransmissions we allow for; moreover, since J e is monotone increasing in k, so 
the optimal performance is obtained for h = t s . As we will see in Section [6j however, the 
co-design decisions are much more involved when we consider closed-loop control over 
multi-hop wireless networks. 
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We can learn several things from this simple analysis: first, it is useful to consider a 
continuous-time performance criterion to be able to compare designs with various sam- 
pling intervals; second, the sampling interval, network latency and packet loss probability 
all impact the achievable closed-loop performance; and third, to solve the co-design prob- 
lem we need to be able to characterize the set of achievable latencies and loss rates on 
a wireless network for a given packet generation rate. 

2.1 Related work 

Further insight into the co-design problem can be obtained by related work in digital 
control, real-time scheduling and networked control. We will briefly review some of the 
work most relevant to ours. 

In many cases, faster sampling gives better control performance, but it also results 
in controllers that are more ill-conditioned and more sensitive to numerical errors that 
arise in fixed-point implementations (see, e.g. [5]). Faster sampling also consumes more 
computing resources. It is therefore often argued that one should choose the longest sam- 
pling period that gives acceptable performance. These arguments have led to a number 
of well-accepted rules-of-thumb for sample time selection, see e.g., the textbooks [21 [5]. 

In a multi-tasking environment, more periodic tasks could run reliably on the same 
machine if these tasks are run less frequently. The precise number of tasks that can be 
run depends on what scheduling policy is used. The celebrated schedulability analysis 
by Liu and Layland [T3] characterizes the maximum utilization for which all tasks can 
be guaranteed to meet their deadlines under rate monotonic and earliest-deadline first 
scheduling. The natural co-design framework is then to adjust the sampling times of 
controllers to optimize the overall system performance while maintaining schedulability 
guarantees, see e.g. [M1E] for representative work. 

Co-design for networked control systems is more complex, since sampling interval, 
latency distribution and reliability of end-to-end transmissions all influence the achiev- 
able closed-loop performance. Even when these parameters are fixed, the optimal control 
problems have been solved rather recently. Nilsson et al. [16] developed linear quadratic 
Gaussian optimal control for discrete-time linear systems with stochastic (networked- 
induced) delay. Sinopoli et al. [25] developed minimum variance optimal estimators for 
networked systems with packet losses and characterized its performance, while Schanato 
et al. [23J considered linear-quadratic control of linear systems under packet loss and 
established a separation principle under the assumption of reliable and instantaneous 
acknowledgements from actuator to controller. An extension to the case of random 
delays and packet losses was developed in [3J. Similarly, on the networking side it is 
well accepted that wireless communication needs reliable packet delivery [281 EZ] , but 
research on wireless communications with hard per-packet deadline constraint is dif- 
ficult and has appeared only recently. Hou et al. [8] proposes a tractable model for 
deadline-constrained traffic where all packets arrive at the beginning of an interval and 
expire at the end of the interval. The main restriction of this work and its various 
extensions jlQL [9] is that it only deals with a single-hop (star) network topology, while 
most practical deployments of low-power wireless rely on multi-hop communications. 
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In particular, recent communication standards for real-time wireless control, such as 
WirelessHAKF [BJ, ISA-100 p] and IEEE 802.15.4e [I], are converging toward a design 
that combines a multi-hop and multi-path routing with a globally synchronized multi- 
channel Time Division Multiple Access (TDMA). Several recent works [211115] study the 
problem of providing reliable real-time communications over multi-hop network. The 
paper [21] formulates the scheduling problem of multiple deadline-constrained periodic 
data flows in WirelessHART network, and proves the problem is NP-hard. A heuristic 
scheduling algorithm is presented, but it assumes reliable links and thus cannot provide 
guaranteed or maximum deadline-constrained reliability. A practical scheduling mecha- 
nism is proposed in [15] based on routing packets on paths with the minimum number 
of consecutive packet losses. 

Several attempts to develop co-design procedures for wireless control systems have 
appeared in the literature. Early attempts focus on resource-constrained scenarios where 
the amount of bits that can be communicated over a wireless channel during a sam- 
pling interval is limited and needs to be allocated to different control loops |3Uj . or 
assumes that only a single controller can access the communication medium at each 
sampling instant [19] . The paper [J3] includes detailed models of the communication 
system, but considers simple network topologies and is neither modular nor optimal. 
The paper p2] focuses on co-design of contention-based medium access and networked 
estimation and studies the dependencies between the number of contenders, the sam- 
pling interval, and the latency and loss distributions of sensor-estimator communication. 
For WirelessHAET networks, our earlier paper |17] argues for structuring the commu- 
nication schedule into network primitives such as unicast and convergecast and develops 
latency-optimal schedules under the assumption that communication links are reliable. 
In parallel work to this paper, |22] has developed a controller-communication co-design 
approach to calculate sampling intervals of multiple controllers to optimize their overall 
control performance and ensure schedulability of the real-time communication. The co- 
design aspect investigates how the additional latency introduced by heuristic retransmis- 
sion policies (which improve end-to-end reliability) impact the closed-loop performance. 
In our initial work [3], we have developed a framework to decompose the co-design prob- 
lem into two well-defined control and communication subproblems. In this paper, we 
prove the optimality of the proposed co-design, and we thoroughly validate the frame- 
work with extensive numerical examples. 

3 Model and problem formulation 

This section summarizes the model and assumptions under which we are able to develop 
a modular co-design framework with provably optimal performance. 

3.1 Process 

We consider the control of a stochastic linear system 

dx = Axdt + Budt + dv c 
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Figure 2: Networked control system block scheme with timing diagrams for sensor, 
controller and actuator nodes. 



where x G W 1 is the state, u G W 71 is the control signal, A G W lXn and B G ^ nxm are 
the system matrices, and v c is a Wiener process with incremental covariance R^. We 
assume that a noisy measurement of the system output 

y{kh) = Cx(kh) + w{kh) 

is taken every sample period h. Here w(kh) is a discrete-time white noise Gaussian 
process, independent of the disturbance v c , and with zero mean and covariance R w . The 
sensor measurements are time-stamped and sent over an unreliable multi-hop network. 

3.2 Network 

The network consists of a set of nodes N = { 1 , . . . , N} that can receive and transmit 
data to the other nodes. We represent the network topology with a directed acyclic 
graph (DAG) Q = (J\f, C) where the presence of a link (n, m) G C C J\f x J\f means that 
there is a non-zero probability that a packet transmitted by node n can be received and 
correctly decoded by node m. Methods for DAG constructions are out of the scope of 
this paper but can be found in e.g. [29] , 

Nodes are synchronized to the global clock and communication is slotted. Each slot 
is t s milliseconds long and allows for the transmission of a single packet and the reception 
of the associated acknowledgement from the receiver. We assume that packet losses on 
links are independent from losses on other links, and that losses on each link (n, m) G C 
follow a Bernoulli process with loss probability P( njm ). While the sensor data packets 
can be lost, we assume that acknowledgements can not. In this way, a transmitting node 
will always know if the packet was successfully decoded by the intended receiver. Nodes 
do not have access to channel state, only their statistics (i.e. packet loss probabilities 
on their outgoing links). The design problem for the network is to develop a forwarding 
policy 7T that determines if a node should forward a received packet or drop it, and to 
which node it should attempt to transmit. 
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3.3 Controller 



The controller and actuator nodes are assumed to be synchronized to the global clock 
and operate with a fixed lag r < h relative to the sampling times of the sensor. The 
controller uses the information available at times kh + r to compute the control action, 
and the actuator uses zero-order-hold and maintains the same control action between 
the controller updates [kh + r, (k + l)h + r) for k = 1, 2, . . . as seen in Figure [2] 



3.4 Codesign objective 



This paper focuses on co-design of the forwarding policy and the control law to optimize 
a linear quadratic loss 



J = E< 
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(1) 



where the matrices Q% x and Qq are symmetric and positive semi-definite while Q c uu is 
symmetric and positive definite. The expectation is taken over the random process and 
measurement disturbances, as well as over the random packet losses in the network. 

The codesign problem is now one of jointly optimizing the sampling interval h, the 
forwarding policy it used by the network nodes, and the control law for computing the 
actuator commands. It is important to notice that closed-loop loss is stated in continuous 
time to allow for comparison of solutions that use different sampling intervals. 



4 A modular co-design framework 

The key contribution of this paper is the development of an optimal and modular co- 
design framework for linear-quadratic control over unreliable multi-hop networks. The 
framework is modular since it seperates the co-design framework into well-defined and 
meaningful networking and control tasks. It is optimal because it allows to find the 
jointly optimal networking and control design subject to our assumptions and restric- 
tions. In this section, we describe the rationale behind our framework, define the net- 
working and control subproblems and show how they are coordinated. The rest of the 
paper is then devoted to solving these problems, establishing optimally of the overall 
design, and demonstrating the effectiveness of the framework on numerical examples. 
The aim of co-design is to solve 

minimize J 

where J is the closed- loop continuous-time linear-quadratic loss 0, and h, it, u indicates 
that we optimize over the sampling period of the sensor, the forwarding policy used by 
network nodes, and the control algorithm. 

After solving the optimal control problem, we can re- write our optimization problem 
on the form 

minimize J*(V(7T,h),h) (2) 



7 



where T>(7r,h) is the packet delay distribution for the forwarding policy ir when sen- 
sor packets are injected with the rate 1/h. Here, J*(T>(TT,h),h) denotes the optimal 
control loss for the given networking parameters. Although it is possible to compute 
J*(D(7r,h),h) (see e.g. [1]), little is known about how the optimal control performance 
depends on the networking parameters. This limits our ability to efficiently solve the 
networking problem Q to optimality. Moreover, even if we could solve the problem to 
optimality, the networking problem is tightly coupled with the control design task and 
the resulting co-design framework would not be modular. 

It is, however, possible to develop an optimal and modular co-design framework 
when we restrict our attention to the time-triggered control architecture described in 



Section 3.3 Nilsson [16J demonstrated that this control structure is suboptimal and 
that better performance can be obtained by an event-triggered architecture that acts 
immediately as data arrives. However, not only does the time-triggered assumption 
simplify the networking and control design, but it also makes the optimal control perfor- 
mance amendable to analysis and will allow us to establish optimality of our framework. 
For this reason, only the packets that are recieved within r seconds from when they 
were sampled are used by the controller. Let p(7r, h, r) be the probability that packets 
injected at the rate 1/h and forwarded using policy tt arrive at the controller within r 
seconds. The co-design problem then becomes 

minimize J* (r, p(jr, h, r), h) 

h,ir 

where r is a fixed sensor-to-controller delay. To limit the number of free parameters, 
we set the transmission delay r equal to the sampling interval h, which results in the 
following form 

minimize J*(p(ir, h), h) 

Note that for fixed sampling interval, J* now only depends on the reliability of end-to- 
end transmissions p(jT,h). While the networking subproblem is still coupled with the 
control design, we will demonstrate in Section [5 .3| that the optimal control loss is mono- 
tone decreasing in the end-to-end reliability p(n,h). Hence, it is optimal to schedule 
the network so as to maximize the deadline- constrained reliability, i.e. the probabil- 
ity that a packet will arrive at the controller node within a fixed delay. This implies 
that for a given sampling interval h and under the restriction of one-sample delayed 
time-triggered control architectures, the optimal co-designed system is obtained by (a) 
scheduling the network to maximize deadline-constrained reliability, and (b) computing 
the control action using the optimal linear-quadratic controller under packet loss. The 
system configuration that minimizes the continuous-time linear quadratic loss can then 
be obtained by a one-dimensional search to find the optimal sampling interval. 



5 Co-design for linear-quadratic control 

In this section, we will develop an optimal co-design for linear-quadratic control over a 
network where the loss process on individual links follow independent Bernoulli processes. 
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The deadline- constrained maximum-reliability forwarding problem, defined and solved in 
Section 5.1 provides the optimal network operation for a fixed sampling interval. The 
optimal control under independent packet losses is developed in Section |5.2| Finally, in 



Section 5.3 we establish the monotonicity properties of the optimal control cost that 
allows us to conclude optimality of our design framework. 



5.1 Deadline-constrained maximum reliability forwarding 

The data flow from sensor to controller node is periodic, with a new packet being pro- 
duced every h seconds. Since the controller disregards packets that have not been de- 
livered within a sample interval, nodes can drop packets that will not be able to meet 
their deadlines. Hence, there will be at most one sensor packet in the network at any 
point in time. Moreover, since the packet loss processes on links are memoryless, the 
optimal policy for the periodic flow can be constructed by the repeated application of 
the optimal forwarding policy for a single transient packet. In the deadline- constrained 
maximum reliability forwarding problem, we pose a strict deadline of D = j- time slots 
and look for forwarding policies that maximize the probability that the sensor packet is 
delivered to the gateway node (directly connected to the controller) within the deadline. 

To solve this problem, we need to introduce some further notation. We let 0(n) be 
the set of outgoing links of node n and d n be the depth of node n in the DAG (i.e. the 
length of the shortest hop path in the DAG from node n to the gateway) . At time slot 
t, let d = D — t be the number of time slots remaining to the deadline, and let r n (d) 
denote the maximum achievable reliability for delivering a packet from node n to the 
gateway within d time slots. Without loss of generality, we let N be the gateway node 
and define N~ = {1, . . . , N — 1} as the set of all other nodes. 

Now, consider a node n that holds a single packet at t = that should be delivered 
to the gateway ./V within a deadline of D time slots. The goal is to develop a forwarding 
policy that maximizes the probability of delivering the packet on time, and to compute 
the associated deadline-constrained maximum reliability r n (D). By definition, 

r N (d) = l, \/0<d<D. 

The maximum reliability and optimal forwarding policies of the other nodes can be found 
by dynamic programming step by step from d = 1 to d = D with the initial condition 

r„(0) = 0, VnG J\f~ . 

At each step, a node n G M~ holding a packet with d time slots until deadline has two 
choices as shown in Fig. |3j it can withhold its transmission, in which case 

r n {d) = r n (d - 1); (3) 



or it can forward the packet to the neighbor that yields the highest forwarding reliability 
r n (d) = max {(1 -p (nm ))r m (d- 1) +P( n , OT )r n (d- 1)}. (4) 

(n,m)eO(n) 
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Figure 3: An illustration of deriving forwarding decisions at node n with deadline d. 



The maximum forwarding reliability of node n is then the maximum of (§ and @. 
Note that r n (d) is readily computed using information from the previous iteration, d— 1, 
of the dynamic programming recursion. The next theorem characterizes the optimal 
forwarding policy and deadline-constrained maximum reliability. 

Theorem 5.1 The following policy n* is optimal for the deadline- constrained maximum 
reliability forwarding problem: nodes n 6 J\f~ drop packets when d £ [0, d n ) and to 
attempt to transmit the packet to node 



m 



mkJarg max {(1 - p {n ifn) )r m (d - 1) + p {n , m) r n {d - 1)} \ (5) 

[ (n,m)ee>(n) J 



when d G [d n ,D]. The probability that the packet will be delivered at the gateway within 
the next d timeslots, r n (d), is given by 

rM= f0, >fde [0,4.-1]; (6) 

P(n,m* 

)r n (d - 1), if d e [d n ,D]. 

Proof. Note that the theorem only gives one optimal policy out of possibly many. In 
particular, the min operator in ^ is used to get a unique forwarder, and transmitting 
even when the deadline can no longer be met would not increase the reliability. Let us 
now prove our claim. 

First, let d 6 [0, d n — 1]. Since node n is at least d hops from the gateway, the packet 
cannot be delivered on time. Thus, r n (d) = for d < d n and it is optimal to withhold 
from transmitting and simply drop the packet. 

For d £ [d n , D], we now establish that the optimal policy always attempts to transmit. 
To this end, we let rt,{d) denote the maximum probability that the packet arrives at the 
gateway before deadline given that node n attempts to forward the packet when d time 
slots remain. It is equivalent to prove that rl(d) > r n (d - 1) for all d € [d n ,D). We 
then proceed by induction from d = d n . Since the loss probabilities for all links in the 
DAG are strictly less than one, forwarding the packet on any of the shortest paths from 
n to the gateway has a non-zero probability to succeed, so rn(d n ) > 0. On the other 
hand, the probability of meeting the deadline if node n witholds its transmission equals 
r n (dn — 1) = 0. Hence, rh(d n ) > r n (d n — 1), and it is optimal to attempt to forward the 
packet. Suppose that for a given d, we have rn(d) > r n (d—l), and let m' be the optimal 
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node for n to transmit to at this time. Then, 



r l( d + 1) = , max {(l-p (nm ))r m (d) + P( n , m )r„(d)} 

(n,ra)60(n) 

> (1 ~ P(n,m>))r m i(d) + P( n ,m')r n (d) 

> (1-P(n,m'))r m >(d- 1) + P( n ,m')r n (d - 1) = r£(d) = r„(d) 
Therefore, transmitting is optimal also at d + 1. □ 



Theorem 5.1 implies that for any node n, we can compute r n (D) efficiently through 
Eqs. ([5])-([6]) using dynamic programming as in Algorithm]!} The optimal packet forward- 
ing policy it* is composed by the forwarding decision m n (d) for every node n G M with all 
possible remaining time to deadline d G [0, D\. Since the time complexity to compute m* 
is proportional to |0(n)|, the time complexity of the algorithm is 0(DN max n |0(n)|). 



Algorithm 1 Optimal forwarding policy 

r n (d) = Vn G AA, Vd G [0, D]; r N (d) = l VdG[0,L>] 
m n (d) = n Vn G A", Vd G [0, D) 
for d = 1 to D do 

for n = 1 to iV — 1 do 
if d > d n then 

compute m* using Eq. let m n (d) = m*. 
end if 

compute r n (d) using Eq. 
end for 
end for 



The deadline-constrained maximum reliability forwarding problem is rich and can be 
extended in many directions, e.g. to consider more realistic channel models where losses 
are correlated in time and nodes need to estimate channel states, to include the impact 
of power control to influence the packet loss probabilities on links, etc. Such extensions 
are outside the scope of this publication, but under current investigation in parallel 
work [32J. For an optimal solution to our co-design problem, however, the following 
observation is critical. 



Lemma 5.1 When the network operates under the optimal forwarding policy it* com- 



puted in Theorem 5.1 with Dt s = h, the packet loss process from a sensor located at 
node n and generating a new packet every h seconds is Bernoulli with success probability 
P = r n (D). 

Proof: The claim follows from the observation that the optimal policy is memoryless 
and that individual links follow Bernoulli processes. 

Lemma [5 . 1 1 implies packets sent over the network operating under deadline-constrained 
maximum reliability forwarding are subject to a delay of Dt s seconds and Bernoulli-losses 



with success parameter p = r n {D). From this perspective, Theorem 5.1 characterizes the 
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Figure 4: DAG routing topology with a sensor at 6-hop from the gateway. 

achievable latency- reliability pairs (Dt s ,p) for a given network. Figure [4] illustrates this 
tradeoff for sampling times h £ [0, 45]i s with t s = 0.01s and three scenarios in which the 
network becomes increasingly unreliable (by increasing the loss probabilities on links). 



5.2 Linear quadratic Gaussian control for fixed forwarding policy 

Under deadline-constrained maximum reliability forwarding, the network delivers sensor 
packets with a fixed delay of r = Dt s seconds and losses samples independently with 
probability 1 — r n (D). We are interested in controllers that work on the information 
available at times kh + r, hold the control signal constant over intervals [kh + T,(k + 
l)h + t), and use control actions that are optimal in the sense of the linear-quadratic 
loss function Q. 

For notational convenience, we assume that T = Nh for some integer N > 0. Let 
Uk = u(kh) be the control signal computed at time kh + r and applied to the process 
during the time interval [kh + T, (k + l)h + r) and let x& = x(kh). Then, the continuous- 
time loss function ([!]) can be transformed into an equivalent discrete-time loss 
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The optimal control problem is then to compute the control sequence {u k } that 
minimizes the discrete-time loss function. The evolution of the system between sampling 
instants can be described in terms of the extended state vector = col{x k , ut-i} as 



6c+i 



Vk 



^Ah 



I, 



ll ~ T e As dsB 



6 + 



Pk [C Oixm] Cfc + w k 



t T e As dsB 
'mx m 



Uk + 


Inxn 







Vk 



= G 



where v k and w k are zero mean discrete-time Gaussian white noise process with 



E< 



Wk 



R v 
R„ 



(8) 



where R v = L e As R^e A1 s ds. 

Note that u k is not computed until time t = kh + t, at which time is available 
to the controller unless it has been dropped by the network. Hence, the controller has 
access to the following information set when computing u k : 

%k — {yk , £4-1 , T^k} , 

Here, y k = (yk, — , yi) , and U k -\ = (u k -i, . . . , Ui) , while TZ k = (p k , . . . , pi) is the 
realizations of the Bernoulli random variable p k that models successful packet transmis- 
sions. It is important to note that the discrete-time loss has cross-terms even if the 
continuous-time loss function does not. The authors in |23j studied a similar problem 
without cross-terms in the loss function. In what follows, we extend the framework 
of [23] to include the cross-coupling terms in the loss function and derive the optimal 
controller and bound its performance. 



5.2.1 Estimator Design 

As in |23j the Kalman filter is the optimal estimator for our setting. The minimum mean 
square error (MMSE) estimate £ k \ k of £ k given by £ k \ k = E{^| I k } can be computed 

recursively starting from the initial conditions £o|-i = c °l{0n.xi, mx i} arid Prj|-l = Po- 
The innovation step is 

£k+l\k = E{&+1 I Zk] = Hk\k + r Uk (9) 

e k+l\k = Ck+1 - 6c+l|fc = $ e k\k + Gv k (10) 

Pk+\\k — E { e fc+i|fc e fc+i|/cl %k} = ^Pk\k^ T + Rv (ii) 
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where R v = GR V G T and v k is independent from T k , while the correction step is 

6c+l|/c+l = Cfe+l|fc + Pk+\Kk+l{zk+l - C£fc+l|fc) (12) 
e k+l\k+l = 6c+l — 6c+l|fc+l (13) 

-ffc+ilfe+i = Pk+i\k - Pk+iKk+iC P k+ i\ k (14) 
Kk+i — Pk+i\kC T {C P k+ i\ k C T + R w )~ l . (15) 

The following result, similar to [23], characterizes the estimation error covariance matrix. 



Proposition 5.1 The MMSE estimate ^ k \ k of £ k is given by the time-varying Kalman 
filter - (15). The expected value of the covariance matrix can be bounded as 



P k \ k < E p {-P fc | fc } < P k \ k 
where the bounds can be computed iteratively as 

P k±l \ k = <^ klk _ l $ T ^R v -p$P klk _ 1 C T (CP klk ^ 1 CT_^ 

Pk\k = Pk\k-1 ~ pPk\k-lC 7 \CP k \ k _ 1 C T + Rw^CP^-i 

P k +l\k = p)t>Pk\k-i$> T + Rv 

P-k\k = (1 _ P)P.k\k-l 

starting from the initial conditions Pol— l = Pq\—i = Po- When k — > oo, the iterations 
converge to the unique stationary solutions P ro and P ro of the modified algebraic Riccati 
equations 

Poo = $Poo$ T + Rv ~ p$PooC T (CP 00 C T + R^CP^ , (16) 

P o0 =(l-p)*Z x * T + R v . (17) 
5.2.2 Controller Design 

Next, we develop the optimal state feedback control law. 

Proposition 5.2 Consider the aforementioned finite horizon LQG control problem. The 
optimal control law 

u k = - (T T S k+1 T + Z uu )- 1 (r T S k+1 $ + Ej u ) e fc | fc (18) 



is a linear function of the estimated state. The matrix S k evolves according to the 
backward Riccati recursion 

S k = <S> T S k+1 $ + % - (<S> T S k+1 T + E iu )(T T S k+1 T + E wl )- 1 (T T S k + 1 Q + ~J U ) , (19) 
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where ^Ife is the MMSE estimate of the state based on the information setl k computed 



with the Kalman filter ^)-(15). As k — >• oo, the Riccati recursion converges to a unique 
stationary solution Soo satisfying 

Soo = $ r Soc$ + % - ($ r £x,r + s^xr^r + s^-^p^oo* + h[j 

for which the associated stationary controller gain is 

Loo = lim L k = -(r T 5oor + EuuT^^S^ + Hi) . 

k— >oo s 

Proof. To derive the optimal feedback control law and the corresponding value for 
the objective function, we apply dynamic programming. We define the optimal value 
function Vfe(£fc) as follows: 



Vfc(60 - minE{£jH^ + 2^E^ u u k + u^E uu u k + Z/J , (20) 

A t-, r tf T p-i 



Fjv&v) = E{^H 6v| Zjv} , (21) 

where = {N — 1, . . . , 1}. We show that = Vo(£o)- The solution of the Bellman 
equation ( 20 ) with the initial condition plj ) is given by 

V k {ik) = EtilSk&l l k } + c k , (22) 

where the nonnegative matrix S k and the scalar c k are independent of the information set 



X k . In contrast to [23], we allow for cross-coupling terms in the Bellman equation (20), 
which is critical for comparing the control costs under different sampling intervals. Apart 
from this, the proof is similar to the one in [21 [23]. □ 

5.2.3 Optimal control cost 

The loss function of the finite horizon LQG for the networked control system can be 
written as 

JV-l N-l 
Jn(p) =£o T So£o + Ti(S P ) + Tr(S k+1 R v ) + £ Tr(($ T S k+l $ + E^ - S k )E p {P klk }) , 

k=0 k=0 

(23) 

where expectation is taken over a Bernoulli sequence {p k } with E{p k } = p. Since no 
efficient way of computing the expectation is known, one can use the upper and lower 



bounds on E p {P k \ k } given in Proposition 5.1 to compute associated upper and lower 



bounds on the finite- horizon control cost J™ m (p) < J^{p) < L™ ax (p). For the infinite 
horizon case, the bounds become 

rmin A 1;™ rmin 

J °° -nZoN Jn 

= Tr(5oo^) + (1 - p)Tr(($ T 5 00 $ + E^ - S^P^) , (24) 

Tmax A 1:~ rmax 

iV->oo iV 

= TriSooRv) + Tr^S^ + E^ - S^P^ - pP^C 7 (CP^C? + R w y l CPoo)) , 

(25) 
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where the matrices P^, -Poo an d 6*00, are given in Proposition 5.1 and Proposition 5.2 
respectively. 



To sum up, the optimal estimator is a time- varying Kalman filter given by (11), (14) 
and (15), while the control law is a static linear feedback ( |18[ ). The combined perfor- 
mance, in the sense of the continuous-time loss function ([I]), can be bounded as in (24) 
and (25). It is this controller and these performance bounds that we will use in our 



co-design. 

5.3 Optimality of the co-design framework 

It makes intuitive sense that the achievable control loss is increasing with increasing loss 
rates in the network. Next, we will give a formal proof that the achievable loss for a fixed 
sampling interval is indeed increasing in the loss probability, which allows us to conclude 
that the combination of linear-quadratic control under loss and deadline-constrained 
maximum reliability routing provides the optimal co-design for a given sampling interval. 

Lemma 5.2 Consider operators f(X) = §X<$> T + R v , h p (X) = X - P XC T {CXC T + 
R^CX and g p {X) = h p (f(X)). If X' < X and p' > p, then g p >{X') < g p {X). 



Proof. By Lemma 8.1 a,c), g p ,{X') < g p {X') < g p (X). □ 



Theorem 5.2 For given sampling interval h, the optimal control loss J^{p) is monotone 
decreasing in end-to-end reliability p. 

Proof. Our proof relies on a coupling argument [12] on the underlying end-to-end loss 
processes. Specifically, for any two Bernoulli processes {pk} and {p' k } with E{p k } = p 
and E{p' fc } = p' such that p < p' we establish that J^(p') < J%(p)- 

The key idea of coupling is to define the two processes on a common probability space 
on which the analysis is carried out. To this end, let {u k } be a sequence of independent 
random variables with uo k ~ U[0, 1] and define p k = l Wfc <p and p' k = l Wfc <p'- Then {pk} 
and {p' k } are Bernoulli trial processes with probabilities p and p' , respectively. A crucial 
property of our construction is that for every realization {co k }, the associated sequences 
{pk{oJk)} and {p'f,(uJk)} satisfy p' k (uJk) > Pk(^k) for all k. In particular, whenever the 
more reliable sequence has a loss, the less reliable sequence also has a loss. 

Our next step is to show that P' N \ N < Pn\n where P' N i N is the estimation error 
covariance matrix under packet delivery sequence {pk} and initial value _Pq| = Po and 
Pn\n is the estimation error covariance matrix under packet delivery sequence {p' k } 
and the same initial value P |o = ^b- We will establish this claim using induction. 
Clearly, P' k \ k < Pk\k holds for k = 0. Assume that it holds for an arbitrary k. Then, 



since P k+ i\k+i = 9 P (Pk\k) and P' k+1 \ k+1 = 9p'{P'k\ k )^ Lemma 5.2 implies that P' k+1 \ k+1 < 
Pk+i\k+i- Hence, by induction, P' N ^ N < P N \ N . 
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Finally, combining this observation with Lemma 



8.2 



J N (p) = c + E TTrA k P k[k (p) 

Pl,P2,-,PN 

fc=0 

N-l 

= c+ E VTrA fc P fc | fc (pH) 

k=0 

N-l 

<c+ E VTrA fc P^(p'( W )) 



fc=0 



AT-1 

= c+ / E # £ TirAfcPfcifcfr/) = Jj^(p') . 

Plv:PjV fc = Q 

This concludes the proof. □ 



Theorem 5.2 allows us to establish optimality of our co-design framework. 



Theorem 5.3 For given sampling interval, the optimal closed loop performance in the 
sense of the linear-quadratic control loss is obtained by scheduling the network to 
maximize deadline- constrained reliability and computing the control action by the time- 
varying LQG- controller @, @. 



Theorem 5.3 establishes that the optimal co-design can be obtained as follows: we 
sweep over the sampling interval h, schedule the network to maximize on-time delivery 
probability, and estimate the associated closed-loop performance of the time-varying 
LQG controller for the corresponding end-to-end reliability p. The optimal co-design is 
obtained for the sampling interval that attains the minimal closed-loop loss. 

It is natural to ask if one could find the optimal h in a more efficient way than 
by exhaustive search. We will show in the numerical examples that the optimal loss 
as a function of h might exhibit multiple local minima, which indicates that it would 
be challenging to find a universal and efficient way of picking the optimal sampling 
interval. Another issue is that there is no efficient way to compute the expectation of 
the covariance matrix with respect to the loss process apart from e.g. Monte Carlo 
simulation [25]. To overcome this problem, one could replace the true performance 
expression by the upper bound J™ ax (/9) and pick the sampling interval that minimizes 
this upper bound. The next result establishes that J™ ax has the same monotonicity 
properties as Jtr, and we will show in the numerical examples that J™ ax provides a 
good surrogate to the true loss function when it comes to picking the optimal sampling 
interval. 

Lemma 5.3 The upper bound on the control loss J™ ax (p) is monotone decreasing in 
end-to-end reliability p. 

Proof. Note that 

J™ x (p) = c' + TrA 00 h p (P 00 (p)) 
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Sampling interval [ms] 



Figure 5: Comparison of the upper bounds J™ ax for three different cases. 



where P 0O (p) is the stationary solution of the modified algrabraic Riccati equation (16) 
for reliability parameter p. Now, consider two reliability parameters p and p' . By similar 

p' > p im plies that -Poo(p') < P oo(p) and by Lemma 8.1 [b), 



5.2 



arguments as in Theorem 
h p (Poo(p)) < h ( /{P 00 {p 1 )). By Lemma 



8.2 



Tin ax 
• J oo 



< J^ ax (p). Our claim is proven. □ 



6 Numerical Examples 



We are now ready to demonstrate our co-design procedure on numerical examples. To 
this end, we consider the following second-order linear system 



dx 
y(kh) 





-wg 



xdt + 



1 



1 

— 2a(ujo 
I x(kh) +w{kh) 







(jjn 



udt + dv c 



(26) 



has incremental covariance = diag{0.5, 0.5}, and w(kh) has covariance 
4 . Our joint design should minimize ([!]) for Q c xx = diag{2, 1}, Q c xu = O2 



J2xl> 



where v c 
R w = 10 

Quu = !> and Qo = °2x2- 

Initially, we consider the system ( |26[ ) to be unstable with the parameters a = — 1, 
( = 1, and wo = 1. Periodic samples of the output y(kh) are transmitted over the 
multi-hop wireless network of Figure [4] with the same link reliabilities as described in 
Section 5.1 For every h G [6, 45]i s , we compute the optimal forwarding policy and the 
associated deadline-constrained reliability using Theorem 5.1 Then, we discretize the 



control loss function Q, solve the corresponding LQG optimal control problem, and 
evaluate its performance as described in Section 5.2 Figure [5] shows the optimal closed- 
loop control losses for varying sampling intervals under the three network scenarios as 
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Table 1: Selection of sampling interval h in the wireless control system that includes 
communication links with loss probability pi 6 [0.75,0.9] 



c 


w 


h (Rule of Thumbs) 


h* (Optimal sampling interval) 


0.1 




40 - 90 ms 


130 ms 


0.2 


7T 


40 - 100 ms 


120 ms 


0.7 


50 - 120 ms 


60 ms 



introduced earlier in Figure [4} We note that there is no universal sampling interval or 
target end-to-end reliability. The optimal sampling interval ranges from 90ms for the 
most reliable network scenario to 250ms for the least reliable case, corresponding to a 
required end-to-end reliability of 65% and 82%, respectively. As the network becomes 
less reliable, more retransmissions (a longer h*) are required to guarantee a sufficiently 
high reliability which causes the associated control loss to increase. Specifically, the 
optimal control loss (marked with a square in Figure [5J increases about a factor of ten 
from the most to the least reliable network scenarios. 

Now, we consider the case when the system (26) is stable, and £ > 0, a = 1, and 



<^o = tt/v 1 — C 2 - This parameterization is chosen to yield roughly the same optimal 
sampling interval independent of Figure [6] shows how the trade-off curves change when 
we vary the damping of the system poles. Similar to the unstable open-loop systems, 
distinct co-design optima also exist for the open- loop stable systems, but they be come 
less relevant when the system poles are better damped. Furthermore, we compare the 
optimal sampling interval obtained by our co-design framework with the rules-of-thumb 
from [2, pp. 129-130]. As can be seen in Table [TJ the optimal sampling interval is more 
often than not outside the range of sampling intervals proposed by the rule-of-thumb. 

It may appear discomforting that our co-design framework requires sweeping over 
the sampling interval to find the jointly optimal design. However, as shown in Figure [7J 
the optimal closed-loop control cost as a function of sampling interval might exhibit 
multiple local minima, which indicates that it will be hard to circumvent this search in 
general. 

A possible limitation of our design procedure is that we set the maximum transmis- 
sion delay equal to the sampling interval to reduce the search space to a single parame- 
ter. One could imagine that a better performance could be achieved by optimizing over 
both the sampling interval and the transmission delay, which is indeed possible in our 
framework. While we cannot rule out this possibility in general, our experience is that 
the additional complexity of a two-dimensional search does not drastically improve the 
performance. 

Finally, since the true closed-loop performance can only be evaluated using Monte 
Carlo simulations, it is useful to be able to rely on more easily computable upper and 
lower performance bounds, introduced in this paper, in the search for the optimal sam- 
pling interval. Figure [8] compares the upper, lower and true closed-loop performance 
for the co-design under the least reliable network scenario. The upper bound J™ ax be- 
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Sampling interval [ms] 




50 100 150 200 250 300 350 400 450 
Sampling interval [ms] 



Figure 6: Comparision of the upper bounds J^ ax for the system (26) which is parame- 
terized by several damping ratios £ and natural frequencies ujq = n/ y 7 ! — £ 2 . 




150 200 250 300 
Sampling interval [ms] 



450 




100 150 200 250 300 350 400 450 
Sampling interval [ms] 



Figure 7: The upper bound J^ ax for the system (26) with the parameters £ 

W0 = 57T. 



0.02 and 
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Figure 8: The experimental Jjy, upper bound J)^ ax and lower bound J™ n for the mini- 
mum control cost with respect to sampling period. The curves are obtained by averaging 
10000 Monte Carlo simulations for the horizon length T = 500 s, with the arrival se- 
quence pk generated randomly. 



comes quite accurate for sampling intervals h > 250ms, and the sampling interval that 
minimizes the upper bound on the performance is rather close to the optimal sampling 
interval for the true cost. Hence, we believe that the upper bound is a good surrogate for 
the true performance if we need to carry out the co-design with limited computational 
resources. 



7 Conclusions 

We considered the joint design of forwarding policies and controllers for networked con- 
trol loops that use multi-hop wireless communication for transmitting sensor data from 
process to controller. By parameterizing the design problem in terms of the sampling 
rate of the control loop, the co-design problem separates into two well-defined networking 
and control design tasks subproblem, both which admit optimal solutions: the network 
should be operated to maximize the delay-constrained reliability, and the control design 
should optimize closed-loop performance under packet loss. We illustrate how these 
problems can be solved, and how the jointly optimal design can be found by a one- 
dimensional search over the sampling interval. To the best of our knowledge, this is 
the first co-design procedure that covers such a breadth of design parameters while still 
guranteeing to find the joint design with optimal closed-loop performance. 

Several natural extensions of our work can be considered. One is the multi-loop con- 
trol problem, where multiple sensors take measurements with different sampling time and 
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deadline. The corresponding networking task, however, becomes a real-time deadline- 
constrained multi-flow scheduling, which was recently proved to be NP-complete [20j . 
so optimal designs will probably be hard to find. Another extension is to consider the 
optimal co-design of a system when also the actuation commands are forwarded over 
an unreliable wireless network. Finally, other sampling strategies, control structures, 
and performance objectives would also be interesting, challenging and possibly useful 
extensions. 

Acknowledgements. The authors are grateful to Prof. Alexandre Proutiere for intro- 
ducing them to the coupling method, and to Dr. Kin Sou and Dr. Daniel Lehmann for 
valuable comments on an earlier draft of this manuscript. 

8 Appendix 

Lemma 8.1 For operators g p (X) and h p {X), matrices X G S™ and F£§" and scalars 
p, p' G [0, 1], the following facts are true. 

a) X<Y => g p (X) < g p (Y) . 

b) X < Y h p (X) < h p (Y) . 

c) p<p' => g p (X)>g pl {X). 

d) p<p' => h p {X)>h pl {X). 

Proof, a) g p (X) < g p (Y) holds as g p (X) is affine in X. Proof for this can be found 
in [25\ Lemma 1(c)]. 

b) As hp is a special form of g p by setting $ = I and R w = 0, we immediately obtain 
the condition h p (X) < h p (Y). 

c) The detail proof is given in [251 Lemma 1(d)]. 

d) As hp is a special form of g p by setting $ = I and R w = 0, we easily conclude 
h p (X) > h p >{X) from (c). □ 

Lemma 8.2 Suppose FeS" and Z G §+. IfY <Z, then 

Tr(XY) < Tr(XZ) MX G S™ . (27) 

Proof. Since Y < Z, it follows that 

x\yx\ < x\zx^ . 

Since trace is a monotone function on the positive definite matrices, we get 

Tv(XY) < Tr(XZ) . 

□ 
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